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^ (57) Abstract: The present invention provides software for electronically annotating electronic images, such as drawings, pho- 
tographs. video, etc., through the drag-and-drop of annotations from a pre-defined, but extendable. list The annotations are placed 

^ at a user-selected X. Y location on the image, and stored in a searchable database. Thus, they can be searched in order to retrieve, 
organize, group, or display desired electronic images or collections of such images. The annotations may be text, images, sounds. 

^ etc. The invention provides a flexible, easy to learn, rapid, low error rate and satisfying interface for accomplishing such tasks. 



bnsdocio: <wo oiei4-ieAij_> 



PCT/US01/04963 



Title of the Invention : 

Methods for the Electronic Annotation, 
Retrieval, and Use of Electronic Images 

Field of the Invention ; 

5 The present invention relates to software for accomplishing the computer- 

facilitated annotation of electronic images, as well as the retrieval and use of such 
annotated images. 

Cross-Reference to Related Applications : 

This application claims priority to United States Patent Application Serial 
10 No. 60/197.971. which application is herein incorporated by reference in its entirety. 

Background of the Invention : c 

Drawings, sketches, artwork, photographs, and other images have long been 
recognized to have significant commercial and/or personal value. Paradoxically, as 
such images age, they become both more valuable and less valuable. Their 

1 5 increased value reflects their success in capturing events, emotions, people, objects, 
achievements, etc. that have attain enhanced personal, historical or commercial 
significance over the passage of time. The paradoxical diminution of their value 
reflects the inexorable loss of information regarding the people, events, or scenes 
shown in the images that occurs when those having actual knowledge of the depicted 

10 images become unavailable or die. This phenomenon, as applied to personal images 
(e.g., photographs, etc.), has been referred to as the "Personal Photo Paradox'" (or 
"P 3 "). 

One means for addressing the paradoxical diminution in the value of aging 
images is to organize the images into scrapbooks or albums, in which the individual 
15 images are captioned. Adding captions to photographs is. however, a time- 
consuming and error prone task even for professional photographers, editors, 
librarians, curators, and scholars. In many professional applications, photographs 
are worthless unless they are accurately described by date. time, location. 
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photographer, title, recognizable people, etc. Additional descriptions may include 
details about the mechanics of the photograph (for example, film type, print size, 
aperture, shutter speed, owner, copyright information) and/or its contents (keywords 
from controlled vocabularies, topics from a hierarchy, free text descriptions, etc.). 
5 In light of the time required to add such captions, amateur photographers rarely 
systematically arrange or notate their collections, except for an occasional 
handwritten note on the back of a photograph or an envelope containing a collection 
of photographs. 

For those who are serious about adding annotations, the common computer- 

10 based approach is to use database programs, such as Microsoft® Access®, that offer 
"form fill-in" or "free" text boxes and then store the information in a database. Data 
entry is typically done by typing, but many systems permit certain attribute values 
for some fields (for example, black & white or color film) to be selected by the 
individual keeping the collection. Of course, simpler tools that provide free-form 

1 5 input, such as word processors, spreadsheets, and other tools are used in many 
situations. Captions and annotations are often displayed near a photograph on 
screen displays, web pages, and printed versions (see. e.g., U.S. Patent Nos. 
6,1 1 1.586 (Issued August 29. 2000): 5.873.080 (Issued February 16. 1999); 
5,493.677 (Issued February 20. 1996); 5,404.435 (Issued April 4. 1995); 5 J 42,662 

20 (Issued August 25, 1992) and by PCT Application WO00046695A 1 (Published 
August 10, 2000) and European Patent Applications EP00938227A3 (Published 
September 6. 2000) and EPOI006464A2 (Published June 7. 2000). Software 
packages (Kodak PhotoEasy®, IvlGI PhotoSuite®, Aladdin Image AXS®. etc.) and 
web sites (Kodak's photonet. Gatherround.com, shutterfly, etc.) offer modest 

25 facilities to typing in annotations and searching descriptions. 

As photograph library sizes increase, the need and benefit of annotation and 
search capabilities grows. The need to rapidly locate photographs of, for example. 
Bill Clinton meeting with Boris Yeltsin at a European summit held in 1998. is strong 
enough to justify substantial efforts in many news agencies. More difficult searches 
30 such as images depicting "agriculture in developing nations" are harder to satisfy. 
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but many web and database search tools support such searches (Lycos, Corbt's. etc.). 
Query-By-I mage-Content from IBM. is one of many projects that use automated 
techniques to analyze image (http://wwwqbic.aimaden.ibm.com/). Computer vision 
techniques can be helpful in finding photographs by color (sunsets are a typical 
5 example), identifying features (corporate logos or the Washington Monument), or 
textures (such as clouds or trees), but a blend of automated and manual techniques 
may be preferable. Approaches to the non-textual or "content-based" annotation of 
images are described in U.S. Patent Nos. 5,91 1.139 (Issued June 8. 1999), 5.899,999 
(Issued May 4. 1999). and 5.579.471 (Issued November 26. 1996) and PCT 
1 0 Applications WO09935596A I (Published July 1 5, 1 999). WO0993 1 605A I 

(Published June 24. 1999). The non-textual searching of image files is discussed by 
Rui. et al. (1999: "Image Retrieval: "Current Techniques Promising Directions and 
Open Issues," http://citeseer.nj.nec.com/384715.html). and by Chang. S.-F. et al. 
(1997). "Visual Information Retrieval from Large Distributed On-line Repositories." 
1 5 http://citeseer.nj.nec.com/ chang97visual.html). Face recognition research offers 

hope for automated annotation, but commercial progress is slow (R. Chellappa. C.L. 
Wilson and S. Sirohey. "Human and Machine Recognition of Faces: A Survey" 
Proceedings of the IEEE, Vol. 83". pp. 705-740, May 1995: Allan Kuchinsky. Celine 
Pering. Michael L. Creech. Dennis Freeze. Bill Serra. Jacek Gwizdka. "FotoFile: A 
20 Consumer Multimedia Organization and Retrieval System 7 '. Proceedings of ACM 
CH/99 Conference on Human Factors in Computing Systems, 496-503. 1999). 

The annotation of photographs is a variation on previously explored 
problems such as annotation on maps (E. Imhof. "Positioning Names on Maps". The 
American Cartographer. 2, 128-144. 1975: J. Christensen. J. Marks, and S. Shieber. 

25 "An Empirical Study Of Algorithms For Point-Feature Label Placement". ACM 
Transactions on Graphics 14, 3, 203-232, 1995: J. S. Doerschler and H. Freeman. 
"A Rule-Based System For Dense-Map Name Placement". Communications Of The 
ACM 35. I. 68-79. 1992) in which the challenge is to place city, state, river, or lake 
labels close to the features. There is a long history of work on this problem, but new 

30 possibilities emerge because of the dynamics of the computer screen. Such efforts, 
however, are fundamentally different from those involved in the annotation of 
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photographs, video, and other electronic images in which the image aspects (i.e.. the 
objects, people scenes, depictions, etc. that are contained within an electronic image) 
are inherently unrelated to one another. Maps (and electronic timelines, such as 
shown http://chaos I .hvpermart.net/ euvpt/tae.html ) are X.Y (or. in the case of 
5 timelines. X) representations of a surface or construct in which the relative 

placements of all desired annotations are determined solely by the scale of the image 
and the X,Y (or X) coordinate of its center, rather than by inspection and analysis of 
the image. The annotation of maps and timelines is possible using prior art methods 
because their image aspects are inherently related to one another. Annotation is 

1 0 usually seen as an authoring process conducted by specialists and users only chose 
whether to show or hide annotations. Variations on annotation also come from the 
placement of labels on markers in information visualization tasks such as in tree 
structures, such in the hyperbolic tree (John Lamping. Ramana Rao. and Peter 
Pirolli. "A Focus + Context Technique Based On Hyperbolic Geometry For 

1 5 Visualizing Large Hierarchies", Proceedings of ACM CHI95 Conference on Human 
Factors in Computing Systems. New York. 401-408. 1995) or in medical histories, 
such as LifeLines (Jia Li.. Catherine Plaisant. Ben Shneiderman, -'Data Object and 
Label Placement for Information Abundant Visualizations" Workshop on New 
Paradigms in Information Visualization and Manipulation (NPIV9H). ACM. New 

20 York. 41-48, 1998). 

Previous work on electronic image annotation focused on the writing of 
computer programs to make label placements that reduced overlaps (Mark D. Pritt. 
"Method and Apparatus for The Placement of Annotations on A Display without 
Overlap". US Patent 5689717. 1997). but there are many situations in which it is 

25 helpful for users to place labels manually, much like a post-it* note, on documents, 
photographs, maps, diagrams, webpages, etc. Annotation of paper and electronic 
documents by hand is also a much-studied topic with continuing innovations (Bill N. 
Schilit. Gene Golovchinsky. and Morgan N. Price. "Beyond Paper: Supporting 
Active Reading with Free Form Digital Ink Annotations." Proceedings of ACM CHI 

30 98 Conference on Human Factors in Computing Systems, v. I 249-256. 1998). While 
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many systems allow notes co be placed on a document or object, the demands of 
annotating personal photograph libraries are worthy of special study (J Kelly Lee 
and Dana Whitney Wolcott, "Method of Customer Photoprint Annotation", US 
Patent 5757466. 1998). Personal photograph libraries are considered to represent a 
i special case because users are concentrating on the photographs (and may have a 
low interested in the underlying technology), are concerned about the social aspects 
of sharing photographs, and are intermittent users. They seek enjoyment and have 
little patience for form filling or data entry. Personal photograph libraries may have 
from hundreds to tens of thousands of photographs, and organization is. to be 
generous, haphazard. Photographs are sometimes in neat albums, but more often put 
in a drawer or a shoebox. While recent photographs are often on top. shuffling 
through the photographs often leaves them disorganized. Some users will keep 
photographs in the envelopes they got from the photography store, and more 
organized types will label and order them. The annotating of such personal images 
is such a time-consuming, tedious and error-prone data entry task that it discourages 
many archivists and librarians from maximizing the value of their image collections. 
As noted by Rui, et al. ( 1 999: http://citeseer ni.nec. com/18471 5.htmn . the 
perception subjectivity and annotation imprecisencss of text-based image retrieval 
systems may cause unrecoverable mismatches in the later retrieval process. The 
problem is particularly significant for most owners of personal photograph libraries. 

As digital cameras become widespread, users have had to improvise 
organization strategies using hierarchical directory structures, and typing in 
descriptive file and directory names to replace the automatically generated 
photograph file numbers. Some software packages (PhotoSuite. PhotoEasy. etc.) 
enable users to organize photographs into albums and create web pages with 
photographs, but annotation is often impossible or made difficult. Web sites such as 
Kodak's PhotoNet.com, AOL.com, Gatherround.com. etc. enable users to store 
collections of photographs and have discussion groups about the collections, but 
annotation is limited to typing into a caption field. The FotoFile software (Allan 
Kuchinsky. Celine Pering. Michael L. Creech. Dennis Freeze. Bill Serra. Jacek 
Gwizdka, "FotoFile: A Consumer Multimedia Organization and Retrieval System". 
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Proceedings of ACM CHI99 Conference on Human Factors in Computing Systems, 
496-503. 1999) is considered particularly relevant. 

It is an objective of the present invention to provide a computer software 
program that permits both skilled and intermittent users the ability to annotate 
5 electronic images in a manner that is easy and accurate, and which permits the user 
or others to retrieve, organize, additionally annotate display and/or distribute such 
images as desired. 

Summary of the Invention : 

The present invention addresses the problem of annotating commercial 
10 and/or personal electronic images, by providing software that permits users to easily 
accomplish such annotation, through the drag-and-drop of annotations from a pre- 
defined, but extendable, list. The annotations are placed at an X.Y location on the 
image determined by the user, and stored in a searchable database. If the image is 
resized, the X.Y position of the center point of the label is proportionately ad justed. 
1 5 The annotations can be searched in order to retrieve, organize, group, or display 

desired electronic images or groups or collections of such images. The annotations 
may be text, images, logos, web pages, web links, sounds, etc. The invention 
provides a flexible, easy to learn, rapid, low error rate and satisfying interface for 
accomplishing such tasks. 

20 In one embodiment, the software causes a scrolling list of annotations (such 

as personal names, dates, objects, events, etc.) to be displayed in response to a user 
action. The user may then access a desired annotation, ■'drag'" it to a relevant site in 
the image and "drop" it into the image. In an alternative or conjunctive 
embodiment, the software may cause menus and submenus of annotations to be 

25 displayed. A desired annotation can be selected and dropped into a relevant site in 
the image. By allowing users to avoid the need for continually re-keying 
annotations (i.e.. by dragging or selecting annotations from a list or menu and 
dropping them onto an image, the task of notating images can be made faster, easier, 
more reliable, and more appealing. Moreover, since the annotations are entered in a 
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database, searching for all images associated with that annotation (e.g.. photographs 
of a friend or family member) is dramatically simplified. The software of the 
present invention described herein comprises a user interface design and the 
database schema that supports such direct annotation. Once images have been 
annotated, they may be retrieved by merely selecting an annotation from a list of 
annotations, and may then collected, organized, displayed or distributed as desired. 

In detail, the invention comprises a computer software-mediated method for 
annotating an image aspect of an electronic image, wherein the software: 

(A) causes the electronic image containing the image aspect to be 
displayed to a user in a computer-generated interface; 

(B) causes the interface to permit the user to access and query a 
searchable electronic database of textual annotation terms for an 
annotation term, and optionally permitting the user to add a desired 
annotation term to the database: and 

(C) causes an annotation of the database to become associated with the 
image aspect, such that: 

( 1 ) the annotation, in response to a request by a user, is displayed 
positioned so as to permit a user to identify the associated 
image aspect in the displayed image; 

(2 J the annotation association is stored in a second searchable 
electronic database so as to permit a user to retrieve the 
electronic image by querying the second electronic database 
for the annotation. 

The invention additionally provides a computer software-mediated method 
for retrieving an electronic image containing a desired image aspect, wherein the 
image is stored in a computer-managed database of images, and wherein the image 
aspect is associated with an annotation, the association being stored in a searchable 
electronic database, the method comprising: 

(A) querying the searchable electronic database for image aspects that are 
associated with the annotation; 
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(B) retrieving from the database an electronic image containing an image 
aspect associated with the annotation of step (A); 

(C) displaying the retrieved electronic image and the annotation of step 
(A) to a user, in response to the user's request, such that the position 

5 of the annotation permits a user to identify the associated image 

aspect in the displayed image. 

The invention further provides a method for retrieving collections of 
electronic images containing a desired image aspect or related family of image 
aspects, wherein the images are stored in a database of images, and wherein the 
10 image aspects are each associated with an annotation or related family of textual 
annotations, each such the association being stored in a searchable electronic 
database, the method comprising: 

(A) querying the searchable electronic database for image aspects that are 
associated with the annotation or related family of annotations: 
15 (B) retrieving from the database electronic images containing an image 

aspect or related family of image aspects associated with the 
annotation or related family of annotations of step (A): 
(C) displaying the retrieved electronic images and the annotation(s) of 
step (A) to a user, in response to the user's request, such that the 
20 position of the annotation permits the user to identify the associated 

image aspect in each of the displayed images. 

The invention additionally provides the embodiments of such computer 
software-mediated methods, wherein the computer is a personal computer, and the 
wherein the displayed image is displayed on the screen of a computer. 

25 The invention additionally provides the embodiments of such computer 

software-mediated methods, wherein the software permits the user to associate the 
annotation with the image aspect by permitting the user to select, drag and drop the 
annotation to a user-selected location in the displayed electronic image. 
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The invention additionally provides the embodiments of such computer 
software-mediated methocs. . herein the image is selected from the group consisting 
of a photograph, a writing, and a drawing. 

The invention additionally provides the embodiments of such computer 
5 software-mediated methods wherein the image is a photograph and wherein the 
image aspect to be annotated is a person depicted in the photograph. 

The invention further provides an electronic image stored in a database of 
images and displayed at the request of a user, the image containing a desired image 
aspect associated with a annotation, wherein, in response to a user's request, the 
10 image can be displayed with the annotation on the electronic image such that the 
position of the annotation permits the user to identify the associated image aspect in 
the displayed image. 

The invention further provides one or more collections of electronic images 
each such image stored in a database of images and displayed at the request of a 
1 5 user, and each containing a desired image aspect or related family of image aspects, 
associated with an annotation o* related family of textual annotations, wherein in 
response to a user's request, the images or collections thereof can be displayed with 
the annotation on the electronic images such that the position of the annotations 
permits a user to identify the associated image aspect in each displayed image. 

20 The invention further provides a computer system specially adapted by 

software to store one or more electronic images or collections of such images, at the 
request of a user, and each containing a desired image aspect or related family of 
image aspects, associated with an annotation or related family of annotations, 
wherein in response to a user's request, the images or collections thereof can be 

25 displayed with the annotation on the electronic images such that the position of the 
annotation permits a user to identify the associated image aspect in each displayed 
image. 
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The invention additionally provides the embodiments of such computer 
software-mediated methods, images and collections of images, and computer 
systems, wherein each image contains more than one image aspect. 

The invention additionally provides the embodiments of such computer 
5 software-mediated methods, images and collections of images, and computer 
systems, wherein such annotations are textual annotations. 

The invention additionally provides the embodiments of such computer 
software-mediated methods, images and collections of images, and computer 
systems, wherein by identifying the position of the annotation, such computer 
10 software-mediated methods, images and collections of images, and computer 
systems permit a user to distinguish the annotated image aspect from inherently 
unrelated image aspect(s) that may be present in the displayed image(s). 

Brief Description of the Figures : 

Figure 1 illustrates the three windows of the user interface created by a 
preferred embodiment of the invention. 

Figure 2 illustrates the linkage relationships between the various Tables 
employed in a preferred embodiment of the software of the present invention. 

Figures 3-8 illustrate the user interface of the software of the present 
invention. Figure 3 shows the initial state of an electronic photographic image. 
Figure 4 illustrates the list-selection of annotations. Figure 5 shows the dragging of 
the annotation from the list and into the photograph. Figure 6 shows the "dropping" 
of the annotation at a user selected site in the photograph. Figure 7 shows the 
annotation of four individuals. Figure 8 shows the ability of the software of the 
present invention to hide annotations upon user request. 

Figure 9 is a flowchart of the general algorithm of software for 
accomplishing the direct annotation of electronic images. 



20 
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Figure 10 is a flowchart of the general algorithm of software for 
accomplishing the deletion of annotations. 

Figure 11 is a flowchart of the general algorithm of software for 
accomplishing the editing of annotations. 

5 description of the Preform Embodiment 

The present invention relates to computer software for facilitating the 
annotation of one or more desired image aspects present in an electronic ^imaue and 
tor employing such annotations to facilitate the retrieval, organization, collection 
display or distribution of the associated electronic images. 

10 As used herein, the term "computer" is intended to encompass anv electronic 

dev,ce that accepts (i.e.. store, generate, or process) image information in electronic 
form and manipulates such information for a result based upon a sequence of 
instructions. Thus, as used herein, the term computer is intended to encompass not 
only mainframe, web-based, personal and laptop computers, but also personal digital 
I a assistants (PDAs), digital cameras, televisions, video game systems (e - 

P.aystation2(S», video storage systems (e.g.. TiVo®). and telephones, etc. that have 
the capability of accepting such information. The term "software" is intended to 
refer to a set or sequence that directs the manipulation of information by a computer. 

The term "image aspect" as used herein is intended to refer to an individual 
20 scene, object, event, design, etc. that is depicted in an electronic ima.e. For 
example, the image aspects of a photograph of people include the depicted 
individuals (singly, e.g. "John Doe." or collectively, e.g.. "Lecture Attendees"), their 
location, etc. 

The term "electronic image," as used herein is intended to include 
25 photographs, video, drawings, writings, webpages. paintings, holographic images, 
cartoons, sketches, renderings, etc. that can be stored electron icallv. 
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The term '"annotation" the association of text (including a textual hyperlink) 
to an image aspect of an image, such that the image aspect can be retrieved, 
collected, displayed or distributed using the annotation. The software of the present 
invention can optionally display or hide some or all annotations in response to the 
5 instructions of a user. 

Image Annotation 

Figures 3-8 show the process of annotation of images aspects of electronic 
images with respect to an electronic photograph of four people at a conference. In 
Figure 3, the initial state of an electronic photographic image is shown, along with a 

1 0 list of names of individuals that have previously been entered into the database due 
to their presence in other images of the library: In Figure 4. the user selects an 
appropriate name from the presented list, and drags it into the photograph (Figure 
5). The annotation is -'dropped" into the photograph at a location selected by the 
user (Figure 6). thereby causing the software to display the annotated individual as 

1 5 one of the '"People in the photograph." The process may be repeated to annotate 

additional people or other image aspects depicted in the image. Figure 7 shows the 
annotation of four identified individuals. Figure 8 shows an attribute of the 
software of the present invention to permit the "hiding" of annotations. 



20 organized scrolling menu, but it could be implemented as a split, or nested menu 
(Andrew Sears and Ben Shneiderman, "Split Menus: Effectively Using Selection 
Frequency To Organize Menus", ACM Transact ions on Computer-Human 
Interaction I. I. 27-51. 1994). In one embodiment, this would entail having 3-5 of 
the most commonly occurring names in a box. followed by the alphabetical 

25 presentation of the full list. Thus the most frequent names would be always visible 
to allow rapid selection. Alternatively, the names might be listed in hierarchical lists 
(e.g.. family, co-workers, friends, etc.) that when selected would display either the 
entire set of names in the selected category, or 3-5 of the most commonly occurring 
names in a box. followed by the alphabetical presentation of the full list. Name 



In Figures 3-8. the selection list is shown as being an alphabetically 
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completion strategies for npid table navigation would be useful in this application 
When users mouse down en name, the dragging begins and a colored box 
surrounds the name. When users mouse up. the name label is fixed in place, a tone 
is sounded, and the database entry of the X.Y coordinates is stored. The tone gives 
further feedback and reinforces the sense of accomplishment. Further reinforcement 
for annotation can be given by subtly changing the border or other attribute of the 
.mage in an image Col.ection Viewer that is capable of showing a representative 
-mage of each image for each collection being viewed (Figure 1). For example, 
when an image (e.g.. a photograph) gets an annotation, the color of its thumbnail's 
border can be changed from white to green. Users will then be able to see how 
much they have accomplished and which images are still in need of annotation. 

Also as shown in Figures 3-8, the user interface of a preferred embodiment 
of the software of the present invention displays a Show/Hide checkbox that gives 
users control over seeing the image with or without the image aspect annotations. 
Smce the image is shown in a resizable window, the position of the labels changes 
upon resizing to make sure they remain over the same person. In a preferred 
embodiment, a small marker (e.g, ten pixels long) extends from the center of the 
label to allow precise placement of annotations when there manv annotated ima»e 
aspects are close together. The marker can be used to point, for example, at the head 
or body of depicted people, and is especially useful in annotating photographs of 
crowded groups. 

In an alternative preferred embodiment, an automatic annotation mechanism 
is employed so that the users can just use a mouse to click, drag and drop to make 
annotations instead of repetitively typing in text fields. These techniques can be 
further combined with functionality for annotation of single photographs, groups of 
selected photographs, or entire collections of photographs. 

Additional preferred embodiments of the software of the present invention 
include the capacity to resize the labels, change fonts, change colors, or add 
animations. Further embodiments permit collaborative annotation by multiple users 
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vvorking side-by-side (J. Stewart, B. B. Bederson. & A. Druin. "Single Display 
Groupware: A Model for Co-Present Collaboration.". Proceedings of ACM CHI99 
Conference on Human Factors in Computing Systems. 286-293. 1999) or 
independently, to annotate images. The multiple annotations could then be 
5 combined, with appropriate resolution of conflicts. Tools for finding variant 

spellings or switches between last and first names could be optionally employed to 
raise data quality and uniformity. A valuable accelerator is bulk annotation (Allan 
Kuchinsky. Celine Pering. Michael L. Creech. Dennis Freeze, Bill Serra, Jacek 
Gwizdka. "FotoFile: A Consumer Multimedia Organization and Retrieval System". 
1 0 Proceedings of ACM CHI99 Conference on Human Factors in Computing Systems. 
496-503. 1999). in which a group of photographs is selected and then the same label 
is applied to even,- photograph with one action, although individual placement might 
still be needed or desired. 

Of course, annotation by names of people in photographs is only an 
15 illustration of the capability of the software of the present invention. Drag and drop 

annotation can be employed in accordance with the methods of the present invention 

for any kind of object in a photograph (e.g.. car. house, bicycle, mountain, sunset. 

etc.). drawing, project (e.g.. children's homework projects), or painting (brushstroke. 

signature, feature), etc. Annotation about the overall image, such as type of 
20 photograph (portrait, group, landscape), or painting (impressionist, abstract, portrait) 

can be added. The software can be used to indicate the importance or quality of the 

images (e.g.. by annotating the images with colored ribbons, multiple star icons. 

etc.). 

The Database Architecture of the Software of the Present Invention 

25 In detail, the software of the present invention operates using an image 

library database. In a preferred embodiment, the software of the present invention is 
preferably implemented using Microsoft Visual Basic 6.0. however, other 
software(e.g.. Java) may be used. Any database capable of linking tables may be 
employed. Preferably, the image library database is a Microsoft® Access <g> database 
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that is able to link tables of image files with their associated annotations. In a 
preferred embodiment, the software of the present invention comprises an image 
library database that contains five linked tables (Figure 2). Through ODBC, The 
software of the present invention is able to connect to and query the database. The 
5 basic concept is that an image library contains "Collections of Images." and that the 
images contain annotated image aspects. The various linked tables employed by the 
present invention are organized in an Image Library schema. 

In the Image Library schema, the "Collections Table" represents the 
collections of images (photographs, drawings, etc.) with optionally provided 
3 attributes. Such attributes may include collection title, description, keywords, 

starting date, ending date, location, a representative PhotoID, a unique Collection ID. 
(either of which may be any set of alphanumeric or keyboard characters) etc. The 
'•Images Table" is where references (full path and file name) of the images and their 
thumbnails are stored, optionally with important attributes such as the date upon 
which the photograph was taken (or image drawn), event, keywords, location, rating, 
color. locale, and so on. Each image should have a unique reference and images 
with the same references are preferably not allowed to be stored in this table even 
though they have different attribute values. The "Linkage Table" is the connection 
between the Collections Table' and "Images Table". It stores the links between 
collections and images. 

Information about the image aspects (for example, people, etc.) shown in the 
images of the Image Library is stored in an "Image Aspects Table". The attributes 
included in the Image Aspects Table may comprise, for example, the location of the 
scene depicted in the image, the name of geographical or political features depicted 
in the image (e.g., the Washington Monument: Mount Whittier), etc. Where the 
image aspect is a person, the attributes included in the Image Aspects Table may 
comprise, for example, the depicted person's Given (or First) name and Family (or 
Last) name, a unique PersonID (for example a social security number, student ID 
number, corporate employee number, or any set of alphanumeric or keyboard 
characters, etc.). Such a PersonID is desirable since it will facilitate the retrieval and 
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storage of image aspects relating to people having the same first and last name 
(people having the same first and last name are preferably not allowed to be stored 
in Image Aspects Table). 

The Image Aspects Table may optionally be extended to include personal or 
5 professional information such as initial impressions, reviews, reactions of others 

present when the image was made, e-mail addresses for exporting the Image Library 
to others, homepage addresses, resumes, occupations and so on. The "Appearance 
Table" stores the information about which image aspect is in which Image. It serves 
as the linkage between the Images Table and the Image Aspects Table. Possible 
10 attributes for this tabie include AppearancelD. PcrsoniD. PhotolD. and relative (X. 
Y) coordinates (upper left corner is (0.0). lower right is (100.100)) of image aspects 
in the image. 

The design of the Image Library preferably considers three major 
assumptions concerning the Library, its Collections, and its Images. These 
15 assumptions can be classified as follows (the present invention is illustrated using 

people depicted in electronic photographs as examples of image aspects of electronic- 
images): 

Relationship between Collections and Images 

A one-to-many relationship between the Collections Table and the Linkage 
20 Table is preferably set so that a collection can contain multiple images, and a one-to- 
many relationship between the Images Table and the Linkage Table is preferably set 
so that same image can be included in multiple collections. It is also possible that a 
collection may contain the same image multiple times to permit its reappearances in 
a slide or other image presentation. Two collections may have exactly same set of 
25 images, may have overlapping sets of images, or may have non-overlapping sets of 
images. If two images have different path names, they are considered to be different 
images for the purposes of the present invention even though they arc copies of one 
another. 
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Relationship benvei n Images and Image Aspects 

A one-to-many relationship between the Images Table and the Appearance 
Table has been set so that an image can contain multiple image aspects (e.g.. 
multiple persons), and a one-to-many relationship between Image Aspects Table and 
5 Appearance Table has been set so that same image aspect can be included in 

multiple images. Multiple appearances of the same image aspect in the same image 
is not allowed. For image aspects that are people, a composite pair of a Given name 
and a Family name should be unique in the Image Aspect Table. 

Relationship among Library, Collections, and Images 

1 0 Within a library', the same image may be contained in multiple collections 

multiple times, but their attributes and annotations should be the same. 

The software of the present invention may readily be expanded or altered to 
include additional or replacement tables, such as a "People Table" for providing 
information about depicted people, an "Object Table"' for providing information 
1 5 about depicted objects, an "Animals Table" for providing information about 

depicted animals, a Keyword Table, and so on, along with connection tables similar 
to the Appearance Table. With such an Image Library database design, even more 
flexible annotation would be possible. 

The software of the present invention causes the Image Library database to 
20 be updated whenever the direct annotation module causes any information changes. 
To illustrate this attribute of software of the present invention, the Image Library 
database may be considered as comprising five categories. The corresponding 
algorithms and implementation issues involved in updating each such category are 
discussed below with respect to image aspects that are persons and in which the 
25 mages are photographs. It will, however, be understood that such illustration is 

relevant to non-photographic images (such as drawings, sketches, etc.) and to image 
aspects other than people. Figure 9 provides a flowchart of a preferred embodiment 
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of the algorithm of the software of the present invention for accomplishing the direct 
annotation of electronic images. 

Adding a New Name Label/Creating a New Person: 

When users drag a name from "People in Library" listbox and drop it onto a 
5 photograph, the software of the present invention checks to determine whether there 
already exists an Appearance connection between the photograph and the person 
since multiple appearances of the same person in a photograph are not allowed. It a 
conllict occurs, the software of the present invention signals such to the user (e.g.. 
highlighting the existing name label on the photograph) and ignores the drag-and- 

10 drop event with a warning message. If there is no conflict, the software of the 
present invention finds the PersonID and PhotolD. calculates a relative (X. Y) 
position (0< X. Y <100) of the drag-and-drop point on the photograph, and then 
creates a new Appearance record with this information. After adding a new record 
to the Appearance table, the Software of the present invention updates the "People in 

15 this Photograph" listbox and finally creates a name label on the photograph. The 
software of the present invention signals to the user that the label has just been 
inserted (e.g.. the newly added name in the "People in this Photograph" listbox will 
be selected, and accordingly the new name label on the photograph will be 
highlighted, etc.). If the added name label is the first one on the photograph, the 

20 software sends an event to the Collection Viewer to change the border color of the 
corresponding thumbnail (for example, to green) in order to show that the 
photograph now has an annotation. 

The algorithm for creating a new person using the software of the present 
invention is simple. As soon as users type in the first name and last name of a 
25 person in the editbox and press enter, the softw are of the present invention checks to 
determine whether the name already exists in the People table. If so. a warning 
message will be displayed with the name in "People in Library" listbox being 
selected. If not. The software of the present invention creates and adds a new Person 
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record to the People table, and then updates the "People in Library" listbox. selecting 
and highlighting the newly added name. 

Deleting Name Label / Deleting Person: 

When the delete button of the software of the present invention toolbar is 
5 clicked or the delete key is pressed, the software checks to determine whether the 
selected name label already exists. If not. the software of the present invention 
ignores the deleting action. But if it does exist, the software automatically calculates 
the Person ID of the selected name label and the Photo! D. and it searches through the 
Appearance Table to find and delete an Appearance record having those IDs. The 
1 0 software of the present invention updates "People in this Photograph" listbox and 
deletes the name label on the photograph. If the deleted name label was the last one 
on the photograph, the software of the present invention sends an event to the 
Collection Viewer to change the border color of the corresponding thumbnail (e.g.. 
to white), to show that the photograph has no annotation. If the user"s focus is on 
1 5 the "People in Library" listbox and the delete key is pressed. The software of the 
present invention finds the PersonID of the selected name in the listbox. The 
software of the present invention deletes the PersonID from the People table and 
also deletes all the Appearance records containing that PersonID. which results in 
the complete elimination of the name label from the other photographs in the 
20 Photograph Library. Again, the Collection Viewer updates the 4border color of 
thumbnails that no longer have annotations. Figure 10 provides a flowchart of a 
preferred embodiment of the algorithm of the software of the present invention for 
accomplishing the deletion of a direct annotation. 

Editing a Name of Person: 

!5 Users of the software of the present invention can edit a name of person in 

library (e.g., by pressing the edit button of the Photo Viewer toolbar or by just 
double clicking over the selected name in the "People in Library" listbox). When the 
edited name is typed in, the software of the present invention finds and changes the 
corresponding person record from the People table only if there is no duplication of 
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the name in the People table. It also refreshes both the "People in this Photograph" 
and the "People in Library" listboxes. and all the name labels on the current 
photograph. If duplication occurs, the whole editing process will be ignored with a 
warning message. Figure 11 provides a flowchart of a preferred embodiment of the 
5 algorithm of the software of the present invention lor accomplishing the editing of a 
direct annotation. 

Posilioning Name Label: 

The software of the present invention permits a user to change the position of 
a name label by drag-and-dropping the existing label over the photograph. As 

1 0 mentioned before, the X.Y position of the center point of a name label relative to its 
associated image aspect is stored in the corresponding Appearance record. Thus, if 
the image is resized, the X.Y position of the center point of the label is 
proportionately adjusted. In a preferred embodiment, the software of the present 
invention uses a small marker hanging down from the center of the label to allow 

1 5 precise placement. The size and direction (downward) of the marker is fixed, which 
may encumber efforts to distinguish labels when many people appear close together 
in a photograph. Using eccentric labels (Jean-Daniel Fekete Catherine Plaisant. 
"Excentric Labeling: Dynamic Neighborhood Labeling for Data Visualization.". 
Proceedings of ACM CHI99 Conference on Human Factors in Computing Systems. 

20 512-519. 1999) or adding an additional (X. Y) field to the Appearance table to allow 
a longer and directional marker may be used in such circumstances. Alternatively, 
other features of the annotation may be changed, such as the font, font color, font 
hinh lighting, font size, justification, etc. and occlusion among labels in resizing the 
photograph may also be avoided. 

25 Importing People (Image Aspects) Table from other Libraries: 

Retyping the names that already exist in other libraries is very tedious and 
time-consuming job. Therefore, the software of the present invention supports a 
function to import the People (Image Aspects) Table from other libraries. The 
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internal process of importing the People Table is similar to that of creating a new 
person repeatedly. The only thing the software of the present invention should 
handle is checking and eliminating the duplication of a person name. 

While the invention has been described in connection with specific 
embodiments thereof it will be understood that it is capable of further modifications 
and this application is intended to cover any variations, uses, or adaptations of the 
invention following, in general, the principles of the invention and including such 
departures from the present disclosure as come within known or customary practice 
within the art to which the invention pertains and as may be applied to the essential 
features hereinbefore set forth. 
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What is claimed is: 

1. A computer software-mediated method for annotating an image aspect of an 
electronic image, wherein said software: 

(A) causes said electronic image containing said image aspect to be 
displayed to a user in a computer-generated interface; 

(B) causes said interface to permit the user to access and query a 
searchable electronic database of annotations for an annotation term, 
and optionally permitting said user to add a desired annotation term 
to said database: and 

(C) causes an annotation of said database to become associated with said 
image aspect, such that: 

(1 ) the annotation, in response to a request by a user, is displayed 
positioned, so as to permit a user to identify the associated 
image aspect in said displayed image; 

(2) the annotation association is stored in a second searchable 
electronic database so as to permit a user to retrieve said 
electronic image by- querying said second electronic database 
for said annotation. 

2. The computer software-mediated method of claim I. wherein said computer 
is a personal computer, and said wherein said displayed image is displayed 
on the screen of a computer. 

3. The computer software-mediated method of claim I. wherein in (C), said 
software permits said user to associate said annotation with said image aspect 
by permitting said user to select, drag and drop said annotation to a user- 
selected location in the displayed electronic image. 

4. The computer software-mediated method of claim I. wherein said electronic 
image contains more than one image aspect. 
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II. 



The computer software-mediated method of claim 1, wherein said annotation 
is a textual annotation. 

The computer software- mediated method of claim I, wherein said image is 
selected from the group consisting of a photograph, a writing, and a drawing. 

The computer software-mediated method of claim 6, wherein said electronic 
image is a photograph. 

The computer software-mediated method of claim 7. wherein said image 
aspect to be annotated is a person depicted in said photograph. 

The computer software-mediated method of claim 8, wherein said annotation 
is a name of said person depicted in said photograph. 

A computer software-mediated method for retrieving an electronic image 
containing a desired image aspect, wherein said image is stored in a 
computer-managed database of images, and wherein said image aspect is 
associated with an annotation, said association being stored in a searchable 
electronic database, said method comprising: 

(A) querying said searchable electronic database for image aspects that 
are associated with said annotation; 

(B) retrieving from said database an electronic image containing an 
image aspect associated with said annotation of step (A); 

(C) displaying said retrieved electronic image and said annotation of step 
(A) to a user, in response to said user s request, such that the position 
of the annotation permits a user to identify the associated image 
aspect in said displayed image. 

The computer software-mediated method of claim 10, wherein said computer 
is a personal computer, and said wherein in said step (C), said image is 
displayed on the screen of a computer. 
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The computer software-mediated method of claim 10. wherein said software 
permits said user to associate said annotation with said image aspect by 
permitting said user to select, drag and drop said annotation to a user- 
selected location in the displayed electronic image. 

The computer software-mediated method of claim 10. wherein said 
electronic image contains more than one image aspect. 

The computer software-mediated method of claim 10. wherein said 
annotation is a textual annotation. 

The computer software-mediated method of claim 10. wherein said 
electronic image is selected from the group consisting of a photograph, a 
writing, and a drawing. 

The computer software-mediated method of claim 15. wherein said 
electronic image is a photograph. 

The computer software-mediated method of claim 16. wherein said desired 
image aspect is a person depicted in said photograph. 

The computer software-mediated method of claim 17. wherein said 
annotation is a name of said person depicted in said photograph. 

A method for retrieving collections of electronic images containing a desired 
image aspect or related family of image aspects, wherein said images are 
stored in a database of images, and wherein said image aspects are each 
associated with an annotation or related family of annotations, each such said 
association being stored in a searchable electronic database, said method 
comprising: 

(A) querying said searchable electronic database for image aspects that 
are associated with said annotation or related family of annotations: 
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(B) retrieving f om said database electronic images containing an image 
aspect or related family of image aspects associated with said 
annotation or related family of annotations of step (A); 

(C) displaying said retrieved electronic images and said annotation(s) of 
step (A) to a user, in response to said user's request, such that the 
position of the annotation permits said user to identify the associated 
image aspect. 

The computer software-mediated method of claim 19, wherein said computer 
is a personal computer, and said wherein in said step (C). said image is 
displayed on the screen of a computer. 

The computer software-mediated method of claim 19. wherein said software 
permits said user to associate said annotation with said image aspect by 
permitting said user to select, drag and drop said annotation to a user- 
selected location in the displayed electronic image. 

The computer software-mediated method of claim 19. wherein said 
electronic image contains. more than one image aspect. 

23. The computer software-mediated method of claim 19. wherein said 
annotation is a textual annotation. 

24. The computer software-mediated method of claim 19. wherein said 
electronic image is selected from the group consisting of a photograph, a 
writing, and a drawing. 

25. The computer software-mediated method of claim 24. wherein said 
electronic image is a photograph. 

26. The computer software-mediated method of claim 25. wherein said desired 
image aspect is a person depicted in said photograph. 

27. The computer software-mediated method of claim 26. wherein said 
annotation is a name of said person depicted in said photograph. 
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An electronic image stored in a database of images and displayed at the 
request of a user, said image containing a desired image aspect associated 
with an annotation, wherein, in response to a user's request, said image can 
be displayed with said annotation on said electronic image such that the 
position of the annotation permits said user to identify the associated image 
aspect. 

The electronic image of claim 28. wherein said annotation was associated 
with said image aspect by having been selected, dragged and dropped into its 
position in the displayed electronic image. 

The electronic image of claim 28. wherein said annotation is a textual 
annotation. 

A collection of electronic images each stored in a database of images and 
displayed at the request of a user, and each such image containing a desired 
image aspect or related family of image aspects, associated with an 
annotation or related family of annotations, wherein in response to a user's 
request, said images or collections thereof can be displayed with said 
annotation on said electronic images such that the position of the annotations 
permits a user to identify the associated image aspect. 

The collection of electronic images of claim 31. wherein said annotation was 
associated with said desired image aspect or related family of image aspects 
by having been selected, dragged and dropped into its respective location in 
the displayed electronic image. 

The electronic image of claim 31. wherein said annotation is a textual 
annotation. 

A computer system specially adapted by software to store one or more 
electronic images or collections of such images, at the request of a user, and 
each containing a desired image aspect or related family of image aspects, 
associated with an annotation or related family of annotations, wherein in 
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response to a user's request, said images or collections thereof can be 
displayed with said annotation on said electronic images such that the 
position of the annotation permits a user to identify the associated image 
aspect. 

The computer system of claim 34. wherein said annotation or related family 
of annotations was associated with said desired image aspect or related 
family of image aspects by having been selected, dragged and dropped into 
its respective location in the displayed electronic image. 

The electronic image of claim 34. wherein said annotation is a textual 
annotation. 
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